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Preface 


The  purpose  of  this  study  was  to  investigate  the  manner 
in  which  Air  Force  Base  Civil  Engineers  (BCEs)  in  the  United 
States  select  architectural-engineering  firms  (A-Es)  for  the 
procurement  of  design  services.  A  review  of  the  A-E 
selection  criteria  used  by  the  BCEs  indicated  that  differing 
interpretations  of  Air  Force  guidance  exist.  The  study 
concluded  with  an  A-E  selection  scoresheet  depicting  the 
aggregated  views  of  the  BCEs .  The  scoresheet  included  the 
selection  criteria  used  most  frequently  and  considered  most 
important  by  BCEs  to  the  overall  goal  of  hiring  highly 
qualified  firms  on  a  fair  basis. 

In  conducting  this  research  and  writing  its  report  of 
findings,  I  have  benefited  greatly  from  the  learned  advice 
of  others.  First,  and  foremost,  I  am  deeply  indebted  to  my 
thesis  advisor,  Captain  Carl  Davis,  for  his  enthusiastic 
support,  timely  feedback,  and  especially  his  vast  experience 
in  research  methods  which  served  to  guide  me  along  a 
logical,  coherent  path  of  study.  I  also  owe  a  word  of 
thanks  to  my  thesis  reader.  Lieutenant  Colonel  John  Ballard, 
for  ably  overseeing  the  progress  of  the  thesis  and  providing 
valuable  insights. 


Mark  S.  Tissi 
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Abstract 

This  research  examined  the  Air  Force  Civil  Engineering 
process  of  selecting  architectural-engineering  firms  (A-Es) 
for  the  procurement  of  facility  design  services.  Sixty-two 
base  civil  engineers  (BCEs)  in  35  states  provided  their 
individual  scoresheets  indicating  the  specific  criteria  by 
which  A-Es  are  evaluated  and  the  range  of  possible  scores 
for  each  criterion. 

Descriptive  statistical  methods  were  used  to  determine 
the  criteria  most  commonly  used  by  BCEs  and  those  which  are 
most  heavily  weighted  in  the  scoring  process.  The  analysis 
included  a  comparison  between  the  actual  criteria  used  and 
the  general  categories  of  criteria  outlined  in  Air  Force 
Regulation  (AFR)  88-31.  Finally,  the  research  used 
inferential  statistical  methods  to  identify  differences  in 
the  application  of  selection  criteria  used  by  bases  of 
differing  major  commands  and  geographic  regions. 

The  study’s  findings  included  a  pictorial 
representation  of  the  aggregrated  scoresheets  in  the  form  of 
an  A-E  evaluation  sheet.  The  evaluation  sheet  contained  the 
criteria  identified  as  those  most  frequently  assessed  by 
BCEs  and  weighted  in  proportion  to  their  importance  as 
perceived  by  the  BCEs. 


AN  EXAMINATION  OF  THE  SELECTION  CRITERIA  USED  IN  THE 
ARCHITECT -ENGINEER  SELECTION  PROCESS  OF  AIR  FORCE  BASE 

CIVIL  ENGINEERS 


I .  Introduction 


Overview 

This  chapter  provides  background  information  on  this 
study’s  general  issue,  the  focus  of  the  study,  its  purpose 
and  justification,  its  specific  objective,  and  the  scope  of 
the  study’s  application. 

Background 

In  recent  years,  the  Air  Force  has  increased  its 
reliance  on  architectural-engineering  firms  (A-Es)  for 
professional  services  needed  to  construct  new  facilities  and 
upgrade  existing  facilities.  From  the  writer’s  experience, 
this  reliance  can  be  expected  to  increase  further  as 
reductions  in  the  officer  corps  begin  to  take  effect.  As 
these  reductions  claim  engineering  design  positions  within 
base  civil  engineering  (BCE)  organizations,  the  use  of  A-E 
services  will  be  even  more  necessary  to  meet  the  BCE  mission 
to  replace,  repair,  and  maintain  Air  Force  real  property. 

As  reliance  on  A-E  services  increases,  the  importance 
of  hiring  qualified  and  competent  firms  increases.  "Hiring 
the  right  architect/engineers  (A/E’s)  and  contractors  is  the 
most  important  contribution  a  manager  can  make  to  a  project" 
(12:98).  Therefore,  the  methods  the  Air  Force  uses  to 


narrow  the  field  and  select  a  competent  A-E  greatly  impacts 
the  quality  of  the  final  product  or  service  delivered  by  the 
A-E.  Such  A-E  services  may  take  the  form  of  facility 
design,  supervision  and  inspection,  factfinding  studies, 
surveys,  investigations,  interior  design,  developing  design 
criteria,  preparing  base  comprehensive  plans,  or  preparing 
action  plans  for  hazardous  waste  cleanup  (4:1). 

The  A-E  service  on  which  this  study  focused  is  facility 
design.  Facility  design  refers  to  the  creative  process 
leading  to  the  "production  of  designs,  plans,  drawings, 
estimates,  specifications,  and  investigative  work  needed  to 
execute  a  construction  project"  (4:1).  The  quality  of  this 
service  can  be  gauged  in  different  ways,  but  for  the  purpose 
of  this  study,  quality  is  best  defined  by  Barra  as  giving 
the  customer  "what  is  required,  namely  a  product  or  service 
fit  for  use,  and  doing  this  in  such  a  way  that  each  task  is 
done  right  the  first  time"  (2:1). 

Technical  competition  is  the  basis  on  which  Air  Force 
BCEs  must  select  A-Es  from  a  field  of  those  bidding  for  a 
contract.  This  requirement  stems  from  the  Federal 
Acquisition  Regulation  (FAR)  and  Air  Force  Regulation  (AFR) 
88-31,  Selecting  Architect -Engineer  ( A-E )  Firms  for 
Professional  Services  by  Negotiated  Contracts . 

Technical  competition  is  a  method  employed 
to  select  a  contractor  where  the  qualifications 
of  the  contractor  are  of  greater  importance 
than  the  ultimate  price  of  the  contract  or 
where  the  nature  of  the  services  to  be  procured 
make  price  competition  impracticable.  .  .  . 


The  A-E  contracting  procedure  consists  of  four 
steps:  (1)  synopsis,  (2)  preselection,  (3) 

selection,  and  (4)  negotiation  and  award 
[8:24], 

All  A-E  contracts  expected  to  result  in  a  fee  greater 
than  $10,000  must  be  synopsized  in  the  Commerce  Business 
Daily,  a  document  published  five  days  a  week  by  the  U.S. 
Department  of  Commerce  (8:7).  Contracts  expected  to  be  of  a 
lesser  amount  need  only  be  publicized  in  the  local  area  of 
the  project  such  as  in  local  newspapers  and  at  tne  oase ' s 
contracting  office  (CO).  The  project  synopsis  must  include 
a  description  of  the  project’s  scope,  the  factors  by  which 
responding  A-E  firms  will  be  evaluated,  and  the  relative 
order  of  importance  of  the  evaluation  factors  (4:6). 

A  preselection  board  consisting  of  at  least  three 
officer  or  civilian  members  is  convened  no  sooner  than  14 
days  after  the  publication  of  the  synopsis.  This  board 
applies  the  publicized  evaluation  factors  to  A-Es  responding 
to  the  synopsis.  An  A-E  is  considered  responsive  by 
maintaining  a  general  resume  in  the  form  of  the  Standard 
Form  254  at  the  base’s  CO,  or  by  forwarding  that  form  to  the 
CO  (3:5-6).  The  preselection  board  narrows  the  field  of 
responding  A-Es  to  a  list  of  the  six  firms  rated  highest 
based  on  their  evaluation,  and  forwards  this  list  to  a 
selection  board  (4:7). 

The  selection  board  is  also  comprised  of  at  least  three 
members,  none  of  whom  served  on  the  preselection  board  for 
the  contract  to  be  awarded.  This  board  reevaluates  the  six 


highest  firms  based  on  factors  similar  to  those  of  the 
preselection  board.  In  addition,  the  selection  board 
conducts  interviews  with  these  A-Es  to  sharpen  the  accuracy 
of  their  individual  evaluations  of  the  firms.  This  board 
then  ranks  the  top  three  qualified  firms  and  forwards  their 
names  to  the  base’s  CO  (4:9). 

The  FAR  provides  that  the  CO  "shall  negotiate  a 
contract  with  the  highest  qualified  firm  ...  at 
compensation  which  the  [CO]  determines  is  fair  and 
reasonable.  .  .  .  Should  the  [CO]  be  unable  to  negotiate  a 

satisfactory  contract  with  the  firm  considered  to  be  the 
most  qualified  .  .  .  negotiations  with  that  firm  should  be 

formally  terminated.  The  [CO]  should  then  undertake 
negotiations  with  the  second  most  qualified  firm.  Failing 
accord  with  the  second  most  qualified  firm,  the  [CO]  should 
terminate  negotiations.  The  [CO]  should  then  undertake 
negotiations  with  the  third  most  qualified  firm"  (3:16). 
Andrews,  in  quoting  a  1979  United  States  House  of 
Representatives  investigative  report,  noted  that  “less  than 
two  percent  of  contract  negotiations  are  formally  terminated 
with  the  top-ranked  [A-E]  firm"  (1:17).  Hence,  the  A-E 
ranked  first  during  the  preselection  and  selection  steps 
will  almost  always  be  awarded  the  contract  for  design 


service . 


Focus  of  the  Study 

Since  the  major  determinant  for  contract  award  is  the 
evaluation  of  A-E  qualifications,  it  is  the  selection 
process  on  which  this  study  focused.  In  particular,  two 
characteristics  of  the  A-E  selection  process  were  of  primary 
concern.  The  first  was  the  range  of  the  specific  criteria 
used  to  evaluate  bidding  A-Es .  The  FAR  and  AFR  88-31  direct 
that  A~E  selection  criteria  will  include  criteria  oased  on 
the  A-E’s  experience,  capability,  current  workload, 
geographic  location,  status  as  a  small  business  or  minority- 
owned  firm,  and  the  amount  of  previous  federal  contracts. 
While  the  criteria  must  have  a  basis  in  the  above  factors. 
Air  Force  BCEs  are  generally  free  to  choose  the  specific 
means  by  which  these  criteria  are  applied  to  A-Es .  This 
leads  to  a  question  as  to  whether  A-Es  are  evaluated 
similarly  at  different  bases. 

The  second  characteristic  of  concern  was  the  relative 
importance  of  the  specific  criteria  BCEs  use  to  evaluate 
bidding  A-Es.  AFR  88-31  directs  that  each  member  of  the 
preselection  and  selection  boards  will  assign  numerical 
ratings  for  each  of  the  above  listed  categories  of 
criteria.  Each  category  and  the  range  over  which  the 
maximum  possible  rating  must  fall  are  presented  in  Table  1. 
For  example,  a  BCE  may  gauge  an  A-E’s  capability  using 
specific  criteria  which  the  BCE  deems  appropriate.  However, 
the  sum  of  the  maximum  possible  ratings  of  the  criteria 
comprising  the  measure  of  capability  must  be  between  20  and 


t.H.1 


30  points.  This  study  used  the  maximum  possible  ratings  as 
measures  of  the  relative  importance  of  the  specific  criteria 
as  perceived  by  BCEs . 

The  federal  government’s  objective  in  choosing  these 
categories  of  criteria  "is  to  assure  an  equitable 
distribution  of  A-E  contracts  among  qualified  firms, 
including  minority-owned  firms  and  firms  that  have  not  had 
prior  government  contracts"  (3:14). 


Table  1 

AFR  88-31  Categories  of  Selection  Criteria 


Category  of  Criteria 


Maximum  Rating 


Experience 

Capability 

Workload 

Location 

Small/Minority  Business 
Prior  DOD  awards 


20  -  30 
20  -  30 
10  -  20 
5-10 

10%  of  total  allowed 
5-10 


Dollar  value  of  prior  awards  _ 20 

Total  points  possible  66  -  132 


(4:7-8 


Statement  of  Problem 

Since  no  study  could  be  found  which  had  examined  the 
equitability  of  applying  the  selection  criteria  listed  in 
Table  1,  the  problem  for  this  research  was  to  determine 
if  Air  Force  BCEs  in  the  United  States  used  the  guidance 
of  the  FAR  and  AFR  88-31  in  the  same  manner.  That  is,  are 
there  differences  in  the  way  bases  interpret  the  FAR  and 
AFR  88-31? 


.1.  .  'f -  •. 
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Sources  of  Differences 

While  Air  Force  bases  can  be  categorized  in  many  ways, 
there  are  two  characteristics  of  bases  which  may  explain  the 
source  of  differences  which  may  arise.  Consequently,  the 
bases  were  grouped  according  to  these  characteristics  to 
identify  trends  in  the  potentially  various  interpretations 
of  the  FAR  and  AFR  88-31.  One  source  of  difference  was  the 
major  command  (MAJCOM)  to  which  the  BCE  organisation  is 
subordinate.  The  second  characteristic  was  the  A-E 
selection  region  to  which  the  base  belongs.  These  regions 
are  geographically  defined  in  AFR  88-31  as  follows: 


1.  Region  1:  Connecticut,  Delaware,  District  of 

Columbia,  Kentucky,  Maine, 
Massachusetts,  New  Hampshire,  New 
Jersey,  New  York,  Ohio,  Pennsylvania, 
Rhode  Island,  Vermont,  Virginia,  West 
Virginia 

2.  Region  2:  Alabama,  Florida,  Georgia, 

Mississippi,  North  Carolina,  South 
Carolina,  Tennessee 

3.  Region  3:  Colorado,  Illinois,  Indiana,  Iowa, 

Kansas,  Michigan,  Minnesota, 

Missouri,  Nebraska,  North  Dakota, 

South  Dakota,  Wisconsin,  Wyoming 

4.  Region  4:  Arkansas,  Louisiana,  New  Mexico, 

Oklahoma,  Texas 

5.  Region  5:  Arizona,  California,  Idaho,  Montana, 

Nevada,  Oregon,  Washington 

(4:11) 

Purpose 

The  purpose  of  this  study  was  to  investigate  the  manner 
by  which  Air  Force  BCEs  select  A-Es  for  the  procurement  of 
design  services.  This  investigation  sought  to  determine 
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if  any  inconsistencies  existed  throughout  the  Air  Force 
with  respect  to  the  specific  criteria  used  in  A-E 
selection  and  the  weighting  of  these  criteria.  As  a 
result  of  this  investigation,  an  evaluation  scoresheet 
was  developed  that  pictorially  represented  the  way  the 
Air  Force  BCEs  select  A-Es.  Knowledge  of  how  BCEs  select 
A-Es  might  help  Air  Force  leaders  determine  if  fair  and 
consistent  applications  of  selection  criteria  exist, 
date ,  no  research  exists  to  help  maximize  objectivity  and 
minimize  subjectivity  in  A-E  evaluations  through  an 
examination  of  how  BCEs  apply  the  FAR  and  AFR  88-31. 

Specific  Ob .iective 

The  specific  objective  of  this  study  Wd5  to  develop  an 
A-E  evaluation  form  based  upon  how  Air  Force  installations 
located  in  the  continental  United  States  (CONUS)  interpret 
the  FAR  and  AFR  88-31.  The  form  included  the  most 
frequently  used  and  important  criteria  which  have  their 
basis  in  one  of  the  seven  categories  of  criteria  listed  in 
AFR  88-31.  This  study  also  recommended  a  distribution  of 
rating  points  for  each  criterion  used  in  the  selection 
process  based  on  its  importance  relative  to  other  criteria 
in  its  category  and  on  the  permitted  range  of  the  maximum 
rating  defined  by  AFR  88-31 .  It  was  not  the  objective  of 
this  research  to  develop  a  better  evaluation  system. 

Adequate  research  is  not  yet  available  to  determine  the 
quality  of  the  present  selection  process.  However,  the  most 


common  criteria  used  to  select  A-Es  were  identified  to 


assess  the  consistency  with  which  A-E  services  are  procured. 


Investigative  Questions 

The  above  specific  objective  was  met  by  examining  the 
answers  to  the  following  investigative  questions'. 

1.  What  set  of  criteria  do  BCEs  use  to  select 
A-Es  for  design  services? 

2.  What  similarities  and  differences  exist 
between  the  criteria  used  at  bases  snd  the 
categories  of  criteria  in  AFR  88-31  with 
respect  to  both  content  and  relative 
importance? 

3.  What  similarities  and  differences  exist  in  the 
criteria  used  to  select  A-Es  between  bases  of 
MAJCOMs  and  geographic  regions  with  respect 

to  both  content  and  relative  importance? 


Scope 


The  scope  of  this  study  was  the  optimum  set  of  criteria 
applied  in  appropriate  proportions  for  the  selection  of  A-Es 
for  facility  design  in  the  CONUS.  Other  criteria,  such  as 
English  proficiency  and  compliance  with  the  Anti-Mafia  Act, 
may  be  important  outside  the  CONUS,  but  they  would  not 
likely  appear  on  this  study's  proposed  standard  evaluation 
form.  Similarly,  criteria  which  are  deemed  important  within 
CONUS  may  not  have  similar  significance  at  overseas 
locations.  In  addition,  the  criteria  used  to  select  A-Es 


for  facility  design  may  not  be  suitable  for  the  selection  of 
designers  of  systems,  such  as  weapons,  computers,  and 
intelligence  networks. 
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Organization  of  the  Thesis 

The  thesis  is  organized  and  presented  in  accordance 
with  the  model  suggested  in  AFIT’s  Style  Guide  for  Theses 
and  Dissertations . 

Chapter  I  contains  an  introduction  to  the  study 
including  the  background  of  the  issue,  and  a  statement  of 
the  study’s  focus,  problem,  purpose,  specific  objective, 
investigative  questions,  scope,  and  organization. 

Chapter  II  contains  a  review  of  literature  relevant  to 
the  study’s  issue.  Topics  discussed  include  selection 
processes  based  on  price,  as  opposed  to  technical, 
competition,  and  evaluation  factors  other  than  those  iistec 
in  AFR  88-31. 

Chapter  III  contains  the  methodology  employed  to  gather 
information  and  the  design  used  in  the  study  for  data 
analysis . 

Chapter  IV  contains  the  major  findings  and  analysis  of 
the  collected  data  and  answers  the  investigative  questions 
leading  to  the  study’s  specific  objective. 

Chapter  V  presents  an  evaluation  sheet  based  sn  the 
aggregated  scoresheets  of  sampled  bases,  a  summary  of  the 
study,  and  recommendations  for  the  further  research  of  the 
A-E  procurement  process. 


0 


I I .  Literature  Review 

Overview 

This  chapter  compares  and  contrasts  opinions  expressed 
in  recent  literature  on  the  use  of  technical  competition  as 
a  basis  for  A-E  evaluation.  It  also  presents  suggested 
additional  selection  criteria.  This  discussion  provides  a 
framework  for  the  issue  of  DOD  procurement  of  A-E  facility 
design  services. 

Bases  of  A~E  Evaluation 

Technical  competition  is  founded  on  the  principle  that 
the  customer  is  best  served  with  quality  service  through  A-E 
competition  of  their  technical  qualifications.  Architect 
and  engineering  societies  have  continuously  endorsed  this 
basis  and  have  opposed  the  consideration  of  fee  in  the 
selection  process.  These  societies  warn  that  selections 
based  on  price  competition  would  result  in  deterioration  of 
the  quality  of  services  rendered  (8:38).  Prior  to  1972,  the 
architect  and  engineering  industries’  Professional  Code  of 
Ethics  forbade  their  members  from  disclosing  fees  with  their 
offers  to  render  service  in  order  to  prevent  cost 
comparisons  with  other  bidding  firms  (8:42). 

In  1986,  William  Moore  wrote  that  competition  based  on 
A-E  fees  would  seriously  undermine  the  relationship  between 
the  selected  firm  and  the  client  (9:228).  He  noted  that 
A-Es,  now  perceived  as  trusted  advisors  and  technical 
experts  would  be  seen  as  competitive  merchants  with  whom 
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the  principle  of  caveat  emptor  must  apply  (9:228). 

In  1987,  Parks  and  McBride  argued  that  when  price 
competition  is  employed,  the  client  must  unilaterally  define 
the  exact  scope  of  work  so  as  not  to  give  any  one  firm  an 
unfair  advantage.  They  noted  that,  under  this  burden,  the 
client  is  depriving  himself  of  the  most  precious  assets 
offered  by  a  qualified  firm  --  creativity,  technical 
expertise,  and  analytical  abilities  --  in  precisely  the 
project’s  most  important  phase  --  project  definition 
(10:71).  In  terms  of  the  project’s  expense,  a  few  dollars 
of  design  cost  may  be  saved  in  the  short  run,  but 
substantially  higher  costs  of  construction  and  facility 
operation  and  maintenance  will  develop  in  the  long  run 
(10:71) . 

Writing  in  opposition,  Andrews,  in  1980,  stated  that 
government  procurement  of  A-E  services  on  the  basis  of 
technical  competition  invites  fixing  of  prices  by  the  A-E 
industry  and  prevents  a  truly  open  competitive  market 
(1:17).  This  issue  received  Congressional  attention  in  1979 
when  the  U.S.  House  of  Representatives  Committee  on 
Appropriations  on  the  Military  Construction  Appropriations 
Bill  expressed  a  "desire  to  assure  price  be  considered  a 
factor  in  the  selection  of  A-E  firms  but  not  the  sole  or 
primary  factor"  (1:18).  Despite  this  and  other  similar 
proposals,  no  Congressional  action  has  changed  the  1972 
Brooks  Act,  Public  Law  92-582,  which  first  established 
technical  competition  as  the  basis  for  government 


procurement  of  professional  A-E  services  (8:46). 


Enacted  by  Congress  in  October  1972,  the  Brooks  Act 
strengthened  technical  competition  in  the  awards  of  A-E 
design  contracts  by  establishing  two  requirements 
(8:17-18).  First,  the  law  requires  the  federal  government 
to  publicly  announce  its  needs  for  A-E  services,  and  to 
negotiate  for  a  fair  and  reasonable  fee  on  the  basis  of 
demonstrated  competence  and  qualification  (3:15).  Second, 
the  law  requires  the  government  to  conduct  discussions  prior 
to  this  negotiation  with  at  least  three  qualified  and 
interested  firms  regarding  anticipated  design  approaches  for 
the  required  work  (3:16). 

The  law’s  opponents  argue  the  following: 

1.  A-Es  can  now  fix  prices  for  design  services. 

2.  Favoritism  and  political  corruption  can  be 
exercised  without  price  competition. 

3.  A-Es  have  no  incentive  to  lower  prices. 

4.  Price  comparison  of  the  various  proposals  is 
precluded  (8:46-47). 

Supporters  of  technical  competition  and  the  Brooks  Act 
argue  the  following: 

1.  An  A-E’s  technical  qualifications  are  more 
important  than  his  fee. 

2.  Price  consideration  would  dominate  the 
selection  criteria. 

3.  A-Es  would  be  forced  to  cut  corners  on  design 
efforts  to  stay  competitive,  resulting  in 
losses  of  quality,  innovation,  and  creativity. 

4.  The  client  would  have  to  do  a  considerable 
amount  of  the  engineering  work  himself 
(8:47;  10:69). 


POD  position .  Despite  the  warnings  of  potential 
corruption  and  the  possibility  of  dollar  savings,  it  remains 
the  DOD,  and  USAF,  position  that: 

1.  Public  Law  92-582  represents  a  reasonable 
balance  of  the  conflicting  objectives  of  a 
model  A-E  selection  process. 

2.  Price  competition  does  not  enhance  the 
selection  of  the  best  qualified  firm. 

3.  Differences  in  A-E  fees  are  negligible  in 
comparison  to  the  potential  savings  and 
benefits  over  the  life  of  a  project  resulting 
from  the  selection  of  the  most  highly  qualified 
firm,  or  conversely,  the  potential  monetary 
loss  resulting  from  the  selection  of  a  less 
qualified  firm  (8:49;  10:70). 

In  short,  the  DOD  favors  the  existing  basis  of 
technical  competition  over  a  modified  basis  using  A-E  fees, 
or  cost,  as  an  evaluative  criterion  for  selection. 

Additional  Selection  Criteria 

AFR  88-31  addressed  the  following  categories  of 
selection  criteria:  experience,  capability,  workload, 
location,  small/minority  business,  prior  DOD  awards,  and 
dollar  value  of  prior  DOD  awards.  AFR  88-31  directs  that 
the  set  of  selection  criteria  must  include  criteria  from 
each  of  the  above  categories.  What  additional  categories, 
if  any,  should  selection  boards  consider  to  ensure  a 
qualified  firm  is  chosen? 

Smith,  in  1983,  wrote  that  the  selection  criteria 
should  include  A-E  references  that  have  been  directly 
investigated,  and  demonstrations  of  follow-through  abilities 
(11:81).  With  its  vast  communications  network,  it  appears 
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as  though  the  DOD  has  an  opportunity  to  apply  this 


criterion.  Shipyards,  bases,  and  posts  across  the  nation 
could  coordinate  with  one  another  to  check  references’ 
credibilities,  assessments  of  A-E  capabilities,  and  the 
satisfaction  of  the  A-E’s  previous  customers. 

Moore,  in  1986,  suggested  that  clients  choose  A-Es 
which  are  compatible  with  themselves  (9:225).  He  defined 
compatibility  as  a  meshing  of  the  firm's  priorities  such  as 
construction  cost,  high  dependabi lity ,  life-cycle  operating 
costs,  and  future  adaptability.  He  noted  further  that  this 
relationship  is  often  the  product  of  extensive,  two-way 
communication,  and  the  philosophical  characteristics  and 
specialties  unique  to  the  A-E  firm  (9:226). 

Summary 

This  chapter  examined  arguments  in  the  recent 
literature  on  the  use  of  an  A-E’s  fee  as  a  criterion  in 
evaluating  firms  in  a  competitive  selection.  While 
proponents  of  its  use  claim  the  fee  criterion  would  prevent 
price-fixing  and  political  corruption,  DOD’s  position 
remains  that  quality  service  is  best  assured  by  excluding 
consideration  of  an  A-E’s  fee  in  favor  of  the  firm’s 
technical  qualifications.  Its  arguments  include  the 
assertion  that  fee  differentials  are  negligible  compare  to 
the  differences  in  future  operations  and  maintenance  costs 
resulting  from  the  award  to  a  lesser  qualified  firm. 


Additional  criteria  suggested  in  the  literature  include 
consideration  of  references  and  a  measure  of  the  A-E’s 
philosophical  compatibility  with  that  of  the  client. 
References  should  be  investigated  thoroughly  with  an 
emphasis  on  the  ultimate  satisfaction  of  the  A-E’s  past 


clients . 


III.  Methodolo 


Overview 


This  chapter  describes  the  steps  which  were  taken  to 
answer  the  investigative  questions  posed  in  Chapter  I. 
These  questions  were  as  follows: 

1 .  What  set  of  criteria  do  BCEs  use  to  select 
A-Es  for  design  services? 

2.  What  similarities  and  differences  exist, 
between  the  criteria  used  at  bases  and  the 
categories  of  criteria  in  AFR  88-31  with 
respect  to  both  content  and  relative 
importance? 

3.  What  similarities  and  differences  exist  in  the 
criteria  used  to  select  A-Es  between  bases  of 

MAJCOMs  and  geographic  regions  with  respect 
to  both  content  and  relative  importance? 

Specifically,  investigative  questions  1  and  2 
examined  the  equitability  of  applying  the  selection 
criteria  among  CONUS  bases.  Investigative  question  3 
was  answered  so  that  trends  and  preferences  of  MAJCOMs 
and  regions  would  be  identified. 


Data  Collection  Procedures 

Information  to  be  analyzed  in  this  study  was  gathered 
in  response  to  a  letter  sent  to  81  CONUS  BCEs  requesting 
information  related  to  local  policies  on  A-E  selections. 

The  81  BCEs  represent  a  census  of  all  CONUS  Air  Force  BCEs. 
The  complete  list  of  solicited  bases  appears  in  Appendix  A. 
Representation  was  expected  from  at  least  70  percent  of  the 
bases.  The  sample  information  was  then  analyzed  and 
compared,  and  common  elements  were  drawn  from  these 


studies.  The  results  of  this  study  are  expected  to  be 
highly  generalizable  to  the  total  population  of  81  CONUS  Air 
Force  bases. 

The  generalizability  of  results  of  this  study,  or  its 
external  validity,  was  a  fundamentally  important  issue. 
External  validity  is  the  generality,  or  range  of  conditions 
to  which  the  observations  under  study  are  relevant  (6:35). 
Since  maximum  external  validity  was  desired  in  this  researcn 
effort,  a  poll  of  the  entire  population  of  CONUS  Air  Force 
BCEs  was  necessary. 

A  letter  was  sent  to  the  BCEs  of  81  CONUS  Air  Force 
installations.  The  letter  requested  a  copy  of  any  locally 
written  policies  used  in  the  selection  or  evaluation  of 
A-Es.  At  a  minimum,  the  letter  specifically  requested  a 
blank  copy  of  the  scoring  sheet  used  at  the  base  which 
indicates  the  specific  criteria  used  by  the  preselection 
board,  and  the  maximum  possible  points  of  each  criterion. 

The  letter  also  fully  disclosed  the  purpose  of  the  request, 
expressed  appreciation  for  participation,  and  guaranteed 
anonymity  of  all  responding  bases  to  elicit  maximum 
cooperation.  A  copy  of  the  letter  and  the  follow-up  to  it 
appear  in  Appendix  B. 

The  criteria  and  maximum  possible  points  for  each 
criterion  were  extracted  from  each  base’s  response  to  the 
letter.  This  information  was  entered  into  a  database  which 
included  the  base’s  name,  sponsoring  MAJCOM,  and 
geographical  region  of  the  United  States.  The  information 


contained  in  the  database  and  the  method  used  to  group  the 
data  into  the  AFR  88-31  categories  are  located  in  Appendix 
C.  The  appendix  also  contains  operational  definitions  for 
each  specific  criterion.  However,  anonymity  of  all 
respondents  was  respected. 

Data  Analysis  Procedures 

Data  analysis  initially  utilized  descriptive  statistics 
by  arranging  the  responses  to  the  request  for  information 
into  several  tables  which  are  described  later  in  this 
chapter.  Descriptive  statistics  are  the  phase  of  statistics 
whose  objectives  are  the  organization  and  summarization  of 
gathered  data  without  seeking  to  draw  inferences  about  a 
group  larger  than  the  contributors  to  the  data  (5:2). 

Similarities  and  differences  between  MAJCOMs  and 
geographic  regions  were  analyzed  by  the  use  of  one-way 
analysis  of  variance  (ANOVA).  In  its  simplest  form,  ANOVA 
is  used  to  test  the  significance  of  the  differences  between 
the  means  of  a  number  of  different  populations. 

Use  of  ANOVA  required  several  assumptions  to  be  made 
about  the  population  from  which  the  data  was  gathered. 

1 .  The  level  of  the  data  gathered  was  at  least 
interval.  For  example,  this  study  measured  criterion 
importance  by  the  reported  maximum  possible  points  for  the 
criterion.  The  assumption  that  the  data  level  was  interval 
means  that  the  change  in  importance  for  a  criterion  from  10 
to  15  points  was  equal  to  the  change  in  importance  from  15 
to  20  points. 

2.  The  populations  from  which  the  data  were  generated 
are  all  distributed  normally  with  means  and  standard 
deviations  that  may  or  may  not  be  unique. 


3.  The  data  were  gathered  in  a  random  and  independent 
process . 

If  further  analysis  was  required  to  identify 
differences  detected  by  ANOVA,  the  Bonferroni  multiple 
comparison  procedure  was  used.  The  Bonferroni  procedure 
effectively  computes  an  interval  length  as  a  function  of  the 
mean  square  error  of  the  data.  Where  sample  means  differ  by 
more  than  the  value  of  computed  interval,  the  difference  in 
the  true  means  is  statistically  significant.  Where  sample 
means  differ  by  less  than  the  value  of  the  computed 
interval,  the  difference  in  the  true  means  is  insignificant, 
or  the  true  means  are  essentially  equal  (5:358). 

Data  analysis  was  conducted  through  the  use  of  the 
computer  facilities  at  the  Air  Force  Institute  of  Technology 
(AFIT),  Wright-Patterson  Air  Force  Base,  Ohio.  The  SAS 
software  system,  a  statistical  analysis  package  installed  on 
AFIT’s  Classroom  Support  Computer,  contains  procedures 
enabling  the  computation  of  means  and  variances  of  data,  and 
the  testing  by  the  ANOVA  and  Bonferroni  procedures  described 
earlier.  SAS’s  PROC  MEANS  procedure  was  used  for  means  and 
variance  computations,  and  the  PROC  ANOVA  procedure  with  the 
BON  option  was  used  for  hypothesis  testing  (7:49-60). 

Steps  for  Investigative  Question  1 

Investigative  question  1  asked,  "What  set  of  criteria 
do  BCEs  use  to  select  A-Es  for  design  services?"  To  answer 
the  first  investigative  question,  the  data  are  presented  in 
tabular  form  according  to  the  range  of  used  criteria 


reported  and  the  frequency  of  their  use  by  the  responding 
bases.  Additional  data  in  these  tables  includes  the  means 
of  the  maximum  possible  points  attainable  for  each 
criterion.  These  tables  present  the  collective  application 
of  AFR  88-31  guidelines  by  Air  Force  BCEs  in  showing  the 
range  of  criteria  evaluated  and  the  relative  importance  of 
each  criterion. 

Steps  for  Investigative  Question  2 

Investigative  question  2  asked,  "What  similarities  and 
differences  exist  between  the  criteria  used  at  bases  and 
categories  of  criteria  in  AFR  88-31  with  respect  to  both 
content  and  relative  importance?"'  To  answer  the  second 
investigative  question,  tables  similar  to  those  used  for 
investigative  question  1  were  used.  Each  criterion,  its 
maximum  possible  points,  and  the  frequency  of  its  use  were 
placed  in  the  appropriate  AFR  88-31  category  of  criteria 
listed  in  Chapter  1,  or  the  criterion  was  set  aside  as  not 
fitting  any  of  the  seven  categories.  This  step  identified 
criteria  being  used  and  the  relative  importance  of  each 
criterion  within  each  category.  Then,  within  each  category . 
the  maximum  possible  points  were  summed  for  comparison  with 
the  AFR  88-31  range  of  maximum  ratings. 

Steps  for  Investigative  Question  3 

Investigative  question  3  asked,  “What  similarities  and 
differences  exist  in  the  criteria  used  to  select  A-Es 


between  bases  of  MAJCOMs  and  geographic  regions  with  respect 


to  both  content  and  relative  importance?"  To  answer  the 
third  investigative  question,  the  data  are  again  presented 
in  tabular  form  similar  to  that  for  investigative  question 
1.  To  examine  similarities  and  differences  between  MAJCOMs, 
a  table  was  developed  with  criteria  listed  down  the  side, 
and  each  MAJCOM  listed  across  the  top.  The  overall  MAJCOM 
means  for  each  specific  criterion  were  entered  in  the 
table.  Then,  the  CONUS  Air  Force  mean  was  included  in  tne 
column  for  the  criterion.  ANOVA  and  Bonferroni  analyses 
were  then  conducted  to  identify  similarities  and 
differences . 

To  examine  similarities  and  differences  between 
geographic  regions,  the  same  steps  were  followed  as  those 
for  the  comparison  of  MAJCOMs  except  that  the  words 
"geographic  region"  should  replace  the  word  “MAJCOM" 
wherever  it  appears  in  the  steps  above. 

Steps  for  the  Specific  Ob lective 

The  specific  objective  of  this  study  was  to  develop  an 
A-E  evaluation  form  based  upon  the  most  common  criteria  used 
by  BCEs  to  select  A-Es .  To  meet  this  objective,  the  most 
widely  used  and  relatively  important  criteria  were 
identified  in  each  table.  These  criteria  and  their  mean 
maximum  possible  points  were  placed  in  an  appropriate 
AFR  88-31  category  of  criteria.  Finally,  for  each  category 
of  criteria,  the  specific  criteria  were  added  to  a  standard 
evaluation  form  until  the  limit  of  the  AFR  88-31  range  of 
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maximum  rating  for  the  category  was  reached.  The  list  of 
specific  criteria  on  this  form  is  the  criteria  used  at  the 
most  bases  and  considered  most  important  by  those  bases. 


Overview 


IV .  Analysis  of  Data 


The  purpose  of  this  study  was  to  examine  those  factors 
of  architectural-engineering  firms  (A-Es)  which  Air  Force 
Base  Civil  Engineers  (BCEs)  use  to  select  the  most  competent 
interested  A-E.  Additionally,  various  interpretations  of 
AFR  88-31  were  consolidated  into  a  standard  evaluation 
method  suitable  Air  Force-wide  for  the  procurement  of 
facility  design  services.  The  analysis  of  data  concerning 
the  factors  is  presented  in  this  chapter. 

The  chapter  is  divided  into  three  parts:  (a)  a 

nationwide  distribution  of  sample  respondents  is  presented; 
(b)  the  general  characteristics  of  selection  criteria  found 
in  the  sample  of  bases  are  examined;  and  (c)  comparisons 
among  different  MAJCOMs  and  A~E  selection  regions  are 
presented.  Additionally,  the  three  investigative  questions 
posed  in  Chapter  I  and  the  study’s  specific  objective  are 
addressed  separately  based  on  the  responses  provided  by  the 
sample  of  BCEs. 

Distribution  of  Respondents 

Sixty-two,  or  77  percent  of  the  surveyed  bases  in  35 
states  responded  to  the  request  for  information.  A  list  of 
the  respondents  is  presented  in  Appendix  A. 

The  respondents  to  the  request  for  information  were 
clustered  for  the  purpose  of  comparison  into  MAJCOMs  and 
geographic  regions  of  the  United  States.  The  MAJCOMs  were 


Air  Force  Logistics  Command  (AFLC),  Air  Force  Systems 
Command  (AFSL',  Air  Training  Command  (ATC),  Military  Airlift 
Command  (MAC),  Strategic  Air  Command  (SAC),  and  Tactical  Air 
Command  (TAC).  For  ease  of  comparison,  the  geographic 
regions  used  were  those  defined  in  AFR  88-31.  That 
regulation’s  five  regions  are: 

1.  Region  1:  Connecticut,  Delaware,  District  of 

Columbia,  Kentucky.  Maine. 
Massachusetts.  New  Hampshire,  New 
Jersey,  New  York.  Ohio,  Pennsylvania, 
Rhode  Island,  Vermont,  Virginia,  West 
Virginia 

2.  Region  2:  Alabama,  Florida,  Georgia, 

Mississippi,  North  Carolina,  South 
Carolina,  Tennessee 

3.  Region  3:  Colorado,  Illinois,  Indiana,  Iowa, 

Kansas,  Michigan,  Minnesota, 

Missouri,  Nebraska,  North  Dakota, 

South  Dakota,  Wisconsin,  Wyoming 

4.  Region  4:  Arkansas,  Louisiana,  New  Mexico, 

Oklahoma,  Texas 

5.  Region  5:  Arizona,  California,  Idaho,  Montana, 

Nevada,  Oregon,  Washington 

(4:11) 

Table  1  summarized  the  selection  criteria  as  outlined 
in  AFR  88-31.  The  table  is  repeated  below  as  a  convenience 
to  the  reader.  The  seven  categories  of  criteria  listed  must 
be  included  in  the  A-E  evaluation  by  the  preselection  and 
selection  boards  (4:7).  Additional  criteria  based  in  other 
categories  may  be  used  in  the  evaluation  process  as  deemed 
appropriate  by  the  BCE.  The  permissible  range  of  the 
maximum  possible  rating  is  further  evidence  of  the  BCE’s 
flexibility  in  applying  the  guidance  in  AFR  88-31.  For 
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example,  the  sum  of  the  maximum  points  available  from 
specific  criteria  used  to  gauge  an  A-E’s  experience  may  fall 
anywhere  between  20  and  30  points.  Those  specific  criteria 
are  also  chosen  and  weighted  at  the  BCE’s  discretion. 


Category  of  Criteria 

Experience 

Capability 

Workload 

Location 


Range  of  Maximum  Rating 

20  -  30 
20  -  30 
10  -  20 

5  -  lu 


Small/Minority  Business  10%  of  total  allowed 
Prior  DOD  awards  5-10 

Dollar  value  of  prior  awards  _ 20 


Total  points  possible  66  -  132 


(4:7-8) 


The  distribution  of  the  respondents  by  MAJCOM  is  shown 
in  Table  2.  The  largest  response  came  from  SAC  which  was 
also  the  largest  MAJCOM  subset  in  the  study’s  census.  One 
base  was  placed  in  an  unidentified  MAJCOM  to  ensure  its 
anonymity . 

The  distribution  of  the  respondents  by  A-E  selection 
regions  is  shown  in  Table  3.  An  acceptable  level  of 
representation  among  the  respondents  existed  for  each 
region . 


Characteristics  of  Selection  Criteria 

Investigative  question  ! .  The  first  investigative 
question  asked  what  set  of  selection  criteria  is  used  by  Air 
Force  BCEs.  The  two  characteristics  of  A-E  selection 
criteria  on  which  this  study  focused  are  the  maximum 
possible  points  attainable  for  each  criterion  used  and  the 


Table  2 


Response  by  Major  Commands 


MAJCOM 

Number  of  Bases 

Percentage 

AFLC 

6 

9.8 

AFSC 

5 

8.  1 

ATC 

11 

17.7 

MAC 

8 

12.9 

SAC 

19 

30.6 

TAC 

12 

19.3 

OTHER 

1 

1 . 6 

Total 

100 . 0 

Table  3 

Response  by 

A-E 

Selection 

Regions 

Region  Number 

of 

Bases 

Percentage 

1 

8 

13.0 

2 

12 

19.3 

3 

12 

19.3 

4 

15 

24.2 

5 

15 

24.2 

Total 

100.0 

frequency  with  which  each  criterion  is  used  by  BCEs . 

Table  4  presents  summary  statistics  from  all  CONUS 
bases  for  each  criterion  used  to  gauge  A-E  experience . 
Included  are  the  number  of  bases  in  the  sample  using  each 
criterion,  the  percentage  represented  by  that  number,  and 
the  mean  of  the  maximum  possible  points  attainable  for  each 
criterion  among  the  bases  using  the  criterion.  The  average 
maximum  possible  point  value  was  obtained  by  calculating  the 


mean  of  the  maximum  ratings  reported  by  the  bases  using  a 
particular  criterion.  Thirty-two  bases  consider  an  A-E’s 
specialized  experience  with  similar  work  and  28  bases  weigh 
the  base’s  past  experience  with  the  particular  A-E.  These 
figures  were  computed  from  information  sent  by  each  base  in 
the  sample.  While  some  bases  used  more  than  one  specific 
criterion  to  assess  an  A-E’s  experience  qualifications,  all 
62  bases  in  the  study  used  at  least  one  criterion  to  gauge 
this  characteristic.  This  finding  is  indicated  below  by  the 
62  bases  shown  in  the  subtotal  of  the  experience  category. 

Table  4 

Average  Maximum  Ratings  and  Percent  Usage 
of  Experience  Criteria 


Criterion 

Number 
of  Bases 

Percent 
of  Bases 

Average 
Max.  Points 

General  Experience 

30 

48 

25 

Specialized  Experience 

32 

52 

22 

Past  Experience 

28 

45 

15 

Subtotal  of  Experience 

62 

100 

30 

Category 


Tables  5  and  6  similarly  depict  the  manner  in  which  an 
A-E’s  capability  and  workload  are  assessed.  Noteworthy  is 
the  fact  that  while  one  would  expect  all  bases  to  use  some 
measure  of  capability  and  workload  in  their  assessment, 
since  the  two  categories  are  specifically  mentioned  in 


v\  ■% 


Table  5 

Average  Maximum  Ratings  and  Percent  Usage 
of  Capability  Criteria 


Criterion 

Number 
of  Bases 

Percent 
of  Bases 

Average 
Max.  Points 

General  Capability 

20 

32 

28 

Availability  of  Key  Personnel  31 

50 

18 

Technical  Qualifications 

17 

27 

18 

Subtotal  of  Capability 
Category 

58 

94 

24 

Table  6 

Average  Maximum  Ratings  and  Percent  Usage 
of  Workload  Criteria 

Criterion 

Number 
of  Bases 

Percent 
of  Bases 

Average 
Max.  Points 

General  Workload 

28 

45 

16 

Capacity  to  Do  the  Work 
in  Time 

34 

55 

12 

Subtotal  of  Workload 
Category 

57 

92 

16 

AFR  88-31,  a  few  bases  do  not  use  any  measures  of  these 
categories  in  their  evaluations.  This  finding  is  indicated 
by  numbers  less  than  62  in  the  subtotal  of  bases  using  the 
respective  categories.  However,  the  many  who  do  measure 
these  characteristics  use  ratings  within  the  AFR  88-31 
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guidelines.  As  indicated  in  Table  1,  the  sum  of  all 
measures  of  capability  should  range  between  20  and  30,  while 
that  of  workload  should  be  between  10  and  20. 

Table  7  presents  the  summary  statistics  for  the 
remaining  4  AFR  88-31  categories  of  criteria.  Of  the  seven 
categories,  the  degree  of  compliance  with  AFR  88-31  is  least 
with  the  two  involving  prior  DOD  awards.  Seventy-six 
percent  of  the  sampled  bases  considered  the  number  of  pri^r 
DOD  contract  awards,  and  only  63  percent  considered  the 
dollar  value  of  those  awards.  Both  criteria  are  included  in 
the  AFR  88-31  guidance  summarized  in  Table  1 . 

Table  7 

Average  Maximum  Ratings  and  Percent  Usage 


of 

Remaining  AFR 

88-31 

Criteria 

Number 

Percent 

Average 

Criterion 

of 

Bases 

of  Bases 

Max.  Points 

Location  of  Firm 

61 

98 

10 

Small /Minority 

Business 

59 

95 

11 

Number  of  Prior 

DOD  Awards 

47 

76 

8 

Dollar  Value  of 

Prior  Awards 

39 

63 

18 

Table  8  examines  the  statistics  for  6  criteria  which 
did  not  fit  neatly  into  any  of  the  7  AFR  88-31  categories. 
The  regulation  provides  for  categories  of  criteria  which 
must  be  included  in  the  selection  process,  and  permits  BCEs 


to  supplement  the  evaluation  with  additional  criteria  such 
as  those  shown. 

Cost  control  effectiveness  refers  to  the  A-E's  ability 
to  deliver  the  required  work  within  its  budgeted  amount. 
Innovative  design  capability  is  generally  measured  by  the 
board  as  the  number  of  design  competitions  won  by  the  firm 
being  evaluated.  The  other  supplemental  criteria  were  a 
capability  to  inspect  the  construction  as  well  as  design  it, 
the  adequacy  of  the  firm’s  organizational  structure,  its 
demonstrated  use  of  computer-aided  design  or  computer-aided 
manufacturing  technology,  and  the  existence  of  a  toll-free 
telephone  access  to  the  firm. 

Table  9  is  a  compilation  of  tables  4  through  8  and 
represents  the  collective  total  of  specific  criteria  used  by 


Table  8 

Average  Maximum  Ratings  and  Percent  Usage 
of  Additional  Criteria 


Number 

Criterion  of  Bases 

Percent 
of  Bases 

Average 
Max.  Points 

Cost  Control  Effectiveness 

11 

18 

10 

Innovative  Design  Capability 

11 

18 

10 

Construction  Inspection  Capab . 

6 

10 

9 

Organizational  Adequacy 

7 

11 

11 

CAD/CAM  Capability 

1 

2 

5 

Toll-free  Phone 

1 

2 

10 

3 
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BCEs.  This  table  shows  that  the  evaluation  process  does 


vary  among  bases,  particularly  in  the  categories  of  criteria 
involving  experience,  capability,  and  workload.  An  example 
of  this  variety  is  that  28  of  the  sampled  bases  consider  an 
A-E’s  past  experience,  while  32  do  not.  The  other  32  bases 
choose  to  use  other  criteria  to  assess  the  A-E’s  experience 
qualifications.  Bases  are  using  different  specific  criteria 
to  gauge  A~E  qualifications  reflecting  a  possible  lack  of 
specificity  in  AFR  88-31’ s  guidance. 

Table  10 

Comparison  of  Regulatory  and  Actual 
Uses  of  A~E  Selection  Criteria 


I 


AFR  88-31  Category 

Of  Criterion 

Percent 

Frequency  of  Use 

Average 
Max.  Points 

Experience  (20  -  30) 

100 

30 

Capability  (20  -  30) 

94 

24 

Workload  (10  -  20) 

92 

16 

Location  (5  -  10) 

98 

10 

Small/Minority  Bus.  (10%  of 

total)  95 

11 

Prior  DOD  Awards  (5  -  10) 

76 

8 

Dollar  Value  of  (20) 

Prior  Awards 

63 

18 

Investigative  question  2.  The  second  investigative 
question  asked  whether  differences  existed  between  the 
regulation’s  guidelines  and  the  specific  criteria  actually 


Investigative  question  3.  The  third  investigative 
question  of  this  study  examined  the  similarities  and 
differences  in  the  application  of  AFR  88-31  guidelines  of 
A-E  selection  criteria  between  MAJCOMs  and  geographic 
regions.  Table  11  reports  the  mean  values  of  the  maximum 
possible  points  attainable  by  MAJCOM  for  each  of  the  18 
specific  criteria  reportedly  used  by  the  BCEs  responding  to 
the  request  for  information.  Additionally,  the  rightmost 
column  of  Table  11  reflects  the  average  maximum  points  of 
all  62  respondents  as  reported  in  Table  9.  This  enables 
each  MAJCOM’ s  average  maximum  rating  for  each  criterion  to 
be  compared  to  that  of  the  entire  sample.  For  example,  for 
the  AFSC  bases  which  choose  to  consider  an  A-E’s  specialized 
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experience,  the  average  of  their  maximum  ratings  for  that 
criterion  was  17.  The  CONUS  Air  Force  average  was  22. 

Table  12  presents  the  means  of  the  maximum  possible 
points  by  A-E  selection  region  for  the  set  of  specific 
criteria.  As  in  Table  11,  the  CONUS  Air  Force  means  for 
each  criterion  are  also  shown  as  calculated  from  the 
sample’s  responses.  The  average  maximum  rating  aval  iab 
from  a  base  in  region  4  which  chooses  to  consider  an  A -o’ 
capacity  to  do  the  work  in  time  is  13,  as  compared  to  tr.e 
sample’s  average  of  12. 

The  summarization  of  MAJCOM  and  regional  averages  in 
Tables  11  and  12  suggests  there  is  variation  in  applying  Air 
Force  guidance  on  A-E  evaluation  even  among  these  groups. 

The  difference  between  the  averages  of  AFLC  and  ATC,  for 
example,  is  22  in  the  case  of  the  past  experience 
criterion.  However,  no  AFLC  bases  considers  that  specific 
criterion  to  gauge  experience.  Rather,  they  consider  other 
experience-related  specific  criteria. 

The  frequency  of  each  criterion’s  use  was  also  of 
interest  in  this  study.  Table  13  reports  each  criterion's 
frequency  of  use  within  each  MAJCOM,  and  Table  14  present! 
their  frequency  by  A-E  selection  region.  Again,  the 
rightmost  column  of  both  tables  reflects  the  CONUS  Air  Force 
average  computed  from  all  62  respondents.  For  example,  bb 
percent  of  ATC  bases  consider  an  A-E’s  technical 
qualifications,  while  only  27  percent  of  all  the  respondents 
consider  that  criterion.  No  AFLC  or  AFSC  bases  reported 
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using  that  criterion.  SAC  is  the  only  MAJCGM  which  uses  all 
the  reported  criteria.  On  the  other  hand,  AFLC ,  AFSC ,  and 
ATC  bases  use  only  criteria  based  in  the  seven  categories  in 
AFR  88-31.  Regionally,  no  selection  region  is  apparently 
far  from  the  Air  Force  usage  average  indicated  in  the 
rightmost  column  of  Table  14.  However,  as  in  the  cases  of 
Tables  11  and  12,  some  variation  in  the  actual  criteria  used 
does  appear  to  exist. 

While  differences  between  MAJCOMs  and  A-E  selection 
regions  are  evident,  the  use  of  analysis  of  variance  (ANOVA) 
and  Bonferroni  multiple  comparison  tests  can  detect  if  the 
differences  noted  are  statistical ly  significant.  That  is, 
are  the  apparent  differences  minor  in  nature  and  due  to  the 
inherent  randomness  of  any  data  collection  procedure,  or  are 
they  significant  enough  to  warrant  further  attention?  ANOVA 
and  Bonferroni  multiple  comparison  tests  were  conducted  on 
the  means  of  the  maximum  possible  points  attainable  for  each 
criterion  to  investigate  this  question.  The  treatment 
groups  of  interest  were  the  six  MAJCOMs  and  the  five  A-E 
selection  regions. 

Table  15  lists  the  criteria  for  which  significant 
differences  existed  in  the  means  of  the  maximum  possible 
points  between  MAJCOMs  and  geographic  regions.  The  table 
also  indicates  which  MAJCOMs  and  regions  differ 
statistically  for  each  criterion  identified  in  the  ANOVA  and 
Bonferroni  tests.  For  example,  the  3.3  point  difference 
between  the  means  at  ATC  and  TAC  bases  which  use  the 
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Table  15 


Criteria  With  Significantly  Different 
Ratings  Between  MAJCOMs  and  Regions 


Criterion 

Differing 
(alpha  = 

Groups 

0.05) 

Amount  of 
Difference 

MAJCOM  Differences 

Points 

Subtotal  of  Experience 
Category 

ATC  - 

TAC 

16 . 5 

Small/Minority  Business  ATC  - 

ATC  - 

SAC 

TAC 

3 . 3 

2 . 9 

Total  Possible  Points 
at  Base 

AFLC  - 
AFSC  - 

AFSC 

ATC 

46.8 

42.6 

Regional  Differences 

Points 

General  Experience 

2  - 

4 

13.3 

Dollar  Value  of  Prior 

Awards  4  - 

5 

6.2 

Innovative  Design  Capability  1  - 

5 

6.5 

small/minority  business  criterion  was  found  to  be 
statistically  significant.  The  chosen  level  of  statistical 
significance  of  0.05  allows  the  differences  to  be  asserted 
with  95  percent  confidence. 

The  reader  should  note  that  Tables  11  and  12  report 
point  differences  much  greater  than  3.3  which  did  not  appear 
in  Table  15  as  statistically  significant.  This  apparent 
paradox  is  due  to  ANOVA’s  and  Bonferroni’s  sensitivity  to 
the  number  of  values  which  contribute  to  the  means  being 
tested.  For  example,  the  3.3  point  difference  discussed 
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earlier  was  based  on  all  of  the  11  participating  ATC  bases 
and  all  of  the  12  responding  TAC  bases.  This  was  because 
all  of  the  ATC  and  TAC  bases  used  the  small/minority 
business  criterion  under  consideration  by  this  example.  In 
contrast,  consider  the  example  of  the  22  point  difference  in 
the  use  of  past  experience  between  AFLC  and  ATC.  ANOVA  did 
not  report  that  difference  as  significant  because,  as  Table 
13  indicated,  only  45  percent  (5)  of  ATC  bases  and  no  AFLC 
bases  used  that  criterion.  In  this  example,  the  absence  of 
testable  data  prevented  the  statistical  test  from  finding 
the  22  point  difference  significant  at  the  95  percent 
confidence  level. 

The  one  noteworthy  trend  in  Table  15  is  that  ATC  is 
involved  in  the  three  criteria  in  which  there  are  differing 
MAJCOMs.  In  each  case,  ATC  has  the  higher  average  maximum 
rating  of  the  differing  commands.  In  the  particular  case  of 
the  experience  subtotal,  ATC’s  average  maximum  rating  is  39 
whereas  AFR  88-31  recommends  that  the  maximum  rating  not 
exceed  30  points.  ATC’s  average  rating  is  16  5  points 
higher  than  that  of  TAC,  and  7  points  higher  than  the  next 
highest  MAJCOM  (SAC). 


V .  Summary  and  Recommendations 

Overview 

The  specific  objective  of  this  study  was  to  develop  an 
A-E  evaluation  form  based  upon  how  Air  Force  installations 
located  in  the  continental  United  States  (CONUS)  interpret 
the  FAR  and  AFR  88-31. 

This  chapter  aggregates  the  set  of  criteria  which  is 
most  frequently  used  and  most  heavily  weighted  by  BCEs 
comprising  the  sample.  In  addition,  this  chapter  presents  a 
brief  summary  of  the  study’s  major  findings,  and  concludes 
with  recommendations  for  further  research  of  the  A-E 
selection  process. 

Specific  Objective 

Table  16  presents  an  A-E  evaluation  form  based  on  the 
criteria  most  frequently  used  and  most  heavily  weighted  by 
BCEs.  It  was  assumed  that  a  criterion’s  use  and  the  maximum 
possible  rating  which  could  be  achieved  for  the  criterion  is 
a  reasonable  measure  of  the  importance  attributed  to  the 
criterion  by  Air  Force  BCEs.  The  data  in  Table  9, 
summarizing  these  two  characteristics  of  each  criterion, 
were  used  to  develop  Table  16. 

Two  additional  considerations  affected  the  content  of 
the  aggregated  scoresheet  in  Table  16.  First,  the  A-E 
scoresheets  received  in  response  to  the  letter  requesting 
the  BCEs’  input  had  varying  levels  of  detail.  For  example, 


Base  A  indicated  that  their  evaluation  of  an  A-E’s 


capability  included  assessments  of  the  availability  of  key 
personnel  and  the  technical  qualifications  of  the  firm.  In 
contrast,  Base  B’s  scoresheet  indicated  only  that  the  A-E’s 
capability  is  to  be  evaluated  without  any  further  guidance 
on  the  measures  of  capability.  In  such  cases,  the  base’s 
specific  criteria  were  entered  in  a  general  criterion  named 
for  the  category  to  which  it  applied.  Similar  differences 
in  detail  in  the  categories  of  experience  and  workload  were 
encountered.  Appendix  D  contains  representative  responses 
to  the  letters  sent  to  all  CONUS  BCEs .  To  safeguard  the 
anonymity  of  all  respondents,  any  references  to  the  source 
of  Appendix  D’s  examples  have  been  omitted. 

Second,  the  limits  on  a  category  of  criteria  imposed  by 
AFR  88-31  were  observed  in  deriving  the  maximum  possible 
points  for  each  criterion.  Where  more  than  one  criterion 
appears  within  a  category,  the  proper  maximum  possible 
points  for  each  criterion  was  determined  in  proportion  to 
its  respective  average  reported  in  Table  9.  For  example, 
both  specialized  and  past  experience  were  considered  in 
assessing  an  A-E’s  experience  qualifications.  Table  9 
showed  that  52  percent  of  all  respondents  considered 
specialized  experience  at  an  average  rating  of  22  points, 
while  45  percent  used  past  experience  at  an  average  rating 
of  15  points.  However,  some  bases  used  both  criteria. 

Using  these  percentages  and  average  ratings  as  weighting 
factors,  aggregated  ratings  for  these  two  criteria  were 
calculated  as  19  for  specialized  experience  and  11  for  past 
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experience,  resulting  in  the  30-point  subtotal  for  the 
experience  category  which  is  in  compliance  with  AFR  88-31 ’s 
guidance . 


Table  16 

Aggregated  Scoresheet  Based  on  Input 
From  62  CONUS  Bases 


Criterion 


Maximum  Possible  Pom 


Specialized  Experience 

Past  Experience  with 
Similar  Work 

Availability  of  Key  Personnel 

Technical  Qualifications  in 
the  Necessary  Disciplines 

Capacity  to  Do  the  Work 
in  Time 

Location  of  Firm  in  Proximity 
of  the  Jobsite 

Small/Minority  Business 

Number  of  Prior  DOD 
Contracts  Awarded 

Dollar  Value  of  Prior  DOD  Awards 

Cost  Control  Effectiveness 

Innovative  Design  Capability  as 
Evidenced  by  Professional  Awards 

Total  Points  Possible 


This  study  investigated  the  manner  by  which  Air  Force 
BGEs  select  A-Es  for  the  procurement  of  design  services. 
After  providing  background  information  and  outlining  the 
steps  taken  in  the  research  process,  the  study  presented  it 
major  findings  in  answering  the  three  investigative 
questions  posed  in  Chapter  I . 

Table  9  reported  the  set  of  specific  criteria  which 
BCEs  use  to  evaluate  an  A-E’s  qualifications.  In  general, 
the  criteria  are  common  to  most  bases  and  are  applied  in  a 
consistent  manner.  Additionally,  Table  10  showed  that  the 
criteria  used  are  generally  consistent  with  AFR  88-31 
guidelines.  Criteria  used  which  do  not  fit  any  of  the 
AFR  88-31  categories  are  cost  control  effectiveness, 
innovative  design  capability,  construction  inspection 
capability,  organizational  adequacy,  computer-aided  design 
or  manufacturing  capability,  and  toll-free  telephone 
access.  Finally,  Table  15  presented  the  differences  in 
application  of  AFR  88-31  guidelines  which  exist  between 

MAJCOMs  and  the  geographically-defined  A-E  selection  .  .  . 

regions . 

The  study  concluded  with  an  A-E  selection  scoresheet 
aggregated  from  the  scoresheets  of  62  BCEs  to  serve  as  a 
cumulative  version  of  the  various  methods  used  by  CONUS  BCE 
to  apply  the  AFR  88-31  guidelines.  The  aggregated 
evaluation  tool  appeared  in  Table  16. 


Recommendations  for  Further  Research 


In  conducting  the  necessary  research  for  this  thesis, 
the  writer  became  aware  of  several  points  that  future 
researchers  of  the  A-E  selection  process  should  consider. 

First,  it  was  obvious  in  the  study’s  initial  stages  of 
development  that  little  research  had  been  accomplished  on 
the  Air  Force  use  of  A-E  design  services.  The  selection 
process  is  only  one  of  many  topics  related  to  this  type  of 
procurement,  and  future  researchers  should  consider  theses 
in  this  area.  Suggested  topics  include  the  origin  of  the 
categories  of  criteria  listed  in  AFR  88-31.  Why  were  those 
categories  considered  appropriate? 

Second,  one  limitation  of  this  study  was  that  it 
applied  only  to  CONUS  Air  Force  installations. 

Consequently,  the  aggregated  scoresheet  cannot  be  considered 
valid  for  overseas  installations  where  other  selection 
criteria  may  be  more  appropriate.  Future  researchers  should 
investigate  the  criteria  necessary  to  consider  the 
uniqueness  of  A-E  qualifications  at  Alaskan,  European,  or 
Far-  Eastern -b-as-e-s-.- . — . .  -  - - - - - - 

Third,  one  recognized  limitation  of  this  research  was 
that  it  involved  only  A-E  selections  made  by  Air  Force 
units.  Future  research  should  study  the  similarities  and 
differences  between  Air  Force  procedures  and  those  of  the 
Army  and  Navy. 

Fourth,  the  writer  recommends  that  future  research 


assess  the  selection  process  from  the  A-E  perspective.  This 


.study  determined  the  criteria  that  the  Air  Force  considers 
important.  It  is  possible  that  the  private  sector  can 
contribute  much  to  validate  or  modify  the  current  procedure 
by  responding  to  a  survey  investigating  the  criteria  they 
believe  best  ensure  a  fair  and  equitable  selection  of  a 
qualified  firm. 

Fifth,  since  all  bases  in  the  sample  did  not  report 
that  they  use  all  of  the  seven  AFR  38-31  categories  of 
criteria,  a  study  should  examine  the  reasons  for  not.  using 
some  of  the  categories.  Are  there  special  circumstances  at 
selected  bases  which  render  some  categories  inappropriate 
for  a  fair  and  competitive  A-E  selection? 

Finally,  this  research  found  a  degree  of  variance  in 
the  A-E  selection  process.  Follow-on  researchers  should 
investigate  if  that  condition  is  in  the  best  interests  of 
the  Air  Force.  While  a  standardized  procedure  would  promote 
a  perception  of  increased  fairness,  the  uniqueness  of  each 
design  project  and  each  Air  Force  base  may  mitigate  the 
possibility  of  one  scoresheet  for  all  selection  boards. 
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Appendix  A:  Lists  of  Solicited  Bases  and  Respondent 


The  following  is  a  list  of  the  81  bases  that  were 
mailed  the  letter  appearing  in  Appendix  B,  and  represents  a 
census  of  active-duty  CONUS  Air  Force  bases. 

Altus  AFB,  Oklahoma 

Andrews  AFB,  Maryland 

Barksdale  AFB,  Louisiana 

Beale  AFB,  California 

Bergstrom  AFB,  Texas 

Blytheville  AFB,  Arkansas 

Bolling  AFB,  District  of  Columbia 

Brooks  AFB,  Texas 

Cannon  AFB,  New  Mexico 

Carswell  AFB,  Texas 

Castle  AFB,  California 

Chanute  AFB,  Illinois 

Charleston  AFB,  South  Carolina 

Columbus  AFB,  Mississippi 

Davis-Monthan  AFB,  Arizona 

Dover  AFB,  Delaware 

Dyess  AFB,  Texas 

Edwards  AFB,  California 

Eglin  AFB,  Florida 

Ellsworth  AFB,  South  Dakota 

England  AFB,  Louisiana 

F.E.  Warren  AFB,  Wyoming 
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Fairchild  AFB,  Washington 
George  AFB,  California 


Goodfellow  AFB,  Texas 
Grand  Forks  AFB,  North  Dakota 
Griffiss  AFB,  New  York 
Grissom  AFB,  Indiana 
Hanscom  AFB,  Massachusetts 
Hill  AFB,  Utah 
Holloman  AFB,  New  Mexico 
Homestead  AFB,  Florida 
Hurlburt  AFB,  Florida 
K.I.  Sawyer  AFB,  Michigan 
Keesler  AFB,  Mississippi 
Kelly  AFB,  Texas 
Kirtland  AFB,  New  Mexico 
Lackland  AFB,  Texas 
Langley  AFB,  Virginia 
Laughlin  AFB,  Texas 
Little  Rock  AFB,  Arkansas 
Loring  AFB,  Maine 
Lowry  AFB,  Colorado 
Luke  AFB,  Arizona 
MacDill  AFB,  Florida 
Malmstrom  AFB,  Montana 
March  AFB,  California 
Mather  AFB,  California 
Maxwell  AFB,  Alabama 
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McChord  AFB,  Washington 
McClellan  AFB,  California 


: 


En 


McConnell  AFB,  Kansas 


I 
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McGuire  AFB,  New  Jersey 

Minot  AFB,  North  Dakota 

Moody  AFB,  Georgia 

Mountain  Home  AFB,  Idaho 

Myrtle  Beach  AFB,  South  Carolina 

Nellis  AFB,  Nevada 

Norton  AFB,  California 

Offutt  AFB,  Nebraska 

Patrick  AFB,  Florida 

Pease  AFB,  New  Hampshire 

Peterson  AFB,  Colorado 

Plattsburgh  AFB,  New  York 

Pope  AFB,  North  Carolina 

Randolph  AFB,  Texas 

Reese  AFB,  Texas 

Scott  AFB,  Illinois 

Seymour- Johnson  AFB,  North  Carolina 

Shaw  AFB,  South  Carolina 

Sheppard  AFB,  Texas 

Tinker  AFB,  Oklahoma 

Travis  AFB,  California 

Tyndall  AFB,  Florida 

Vance  AFB,  Oklahoma 

Vandenberg  AFB,  California 


■c.> 
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Warner-Rofcins  AFB,  Georgia 
Whiteman  AFB,  Missouri 
Williams  AFB,  Arizona 
Wright-Patterson  AFB,  Ohio 
Wurtsmith  AFB,  Michigan 

The  following  is  a  list  of  the  62  bases  that  responded 
to  the  letters  in  Appendix  B,  and  represents  a  percent 
response  rate. 

Andrews  AFB,  Maryland 
Bergstrom  AFB,  Texas 
Blytneville  AFB,  Arkansas 
Brooks  AFB,  Texas 
Carswell  AFB,  Texas 
Castle  AFB,  California 
Chanute  AFB,  Illinois 
Columbus  AFB,  Mississippi 
Davis-Monthan  AFB,  Arizona 
Dyess  AFB,  Texas 
Edwards  AFB,  California 
Eglin  AFB,  Florida 
Ellsworth  AFB,  South  Dakota 
England  AFB,  Louisiana 
F.E.  Warren  AFB,  Wyoming 
Fairchild  AFB,  Washington 
George  AFB,  California 
Grissom  AFB,  Indiana 


Hanscom  AFB,  Massachusetts 


Hill  AFB,  Utah 


Holloman  AFB,  New  Mexico 


Homestead  AFB,  Florida 


Hurlburt  AFB,  Florida 


K.I.  Sawyer  AFB,  Michigan 


Kelly  AFB,  Texas 


Keesler  AFB,  Mississippi 


Kirtland  AFB,  New  Mexico 


Lackland  AFB,  Texas 


Langley  AFB,  Virginia 


Loring  AFB,  Maine 


Lowry  AFB,  Colorado 


Luke  AFB,  Arizona 


Malmstrom  AFB,  Montana 


Mather  AFB,  California 


Maxwell  AFB,  Alabama 


McChord  AFB,  Washington 


McClellan  AFB,  California 


McConnell  AFB,  Kansas 


McGuire  AFB,  New  Jersey 


Minot  AFB,  North  Dakota 


Myrtle  Beach  AFB,  South  Carolina 


Nellis  AFB,  Nevada 


Offutt  AFB,  Nebraska 


Patrick  AFB,  Florida 


Pease  AFB,  New  Hampshire 


*  A  »’*  , 
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Plattsburgh  AFB,  New  vork 
Pope  AFB,  North  Carolina 
Randolph  AFB,  Texas 
Reese  AFB,  Texas 
Scott  AFB,  Illinois 

Seymour- Johnson  AFB,  North  Carolina 

Shaw  AFB,  South  Carolina 

Sheppard  AFB,  Texas 

Tinker  AFB,  Oklahoma 

Travis  AFB,  California 

Vance  AFB,  Oklahoma 

Vandenberg  AFB,  California 

Warner-Robins  AFB,  Georgia 

Whiteman  AFB,  Missouri 

Wright-Patterson  AFB,  Ohio 

Wurtsmith  AFB,  Michigan 
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Appendix  B'-  Letters  to  CONUS  BCEs 


FROM:  AFIT/LSG  30  Jan  87 

SUBJECT:  Research  of  Architect-Engineer  (A-E) 

Selection  Criteria 

TO:  CES/CC 

1.  I  am  conducting  thesis  research  at  the  Air  Force 
Institute  of  Technology.  I  solicit  your  generous 
cooperation  to  contribute  the  data  necessary  to  the  subject 
research.  In  particular,  I  am  requesting  that  you  forward 
one  unused  copy  of  the  evaluation  form,  or  scoresheet.  used 
by  preselection  boards  at  your  unit  to  select  A-Es  for  tne 
design  of  construction  project.  In  addition,  piease  forward 
a  copy  of  any  local  policies  affecting  A-E  selections  which 
are  in  effect  at  your  base. 

2.  The  characteristics  of  interest  on  the  evaluation  form 
are  the  specific  criteria  scored  by  your  unit  and  the 
numeric  range  within  which  each  specific  criterion’s  score 
may  fall.  These  data  will  be  used  to  identify  trends  within 
and  across  major  commands,  geographic  regions,  and  the  Air 
Force  as  a  whole.  As  a  result  of  this  research,  I  hope  to 
consolidate  these  forms  into  a  product  of  use  to  all 
commands . 

3 .  Your  invaluable  and  timely  support  would  be  greatly 
appreciated.  Your  organization  will  have  complete  anonymity 
throughout  the  thesis  process,  including  the  final  report. 
Upon  completion,  the  results  and  recommendations  of  this 
research  will  be  available  upon  request  to  AFIT. 

!  4.  For  more  information  concerning  my  request,  please  call 

me  at  AUTOVON  785-5435.  Please  forward  your  unit’s  input 
via  official  distribution  to: 

1st  Lt  Mark  S.  Tissi 
AFIT/LSG 
WPAFB  OH  45433 


MARK  S.  TISSI,  1st  Lt ,  USAF 

Graduate  Student  in  Engineering  Management 

Air  Force  Institute  of  Technology 


FROM:  AFIT/LSG 


2  Mar  87 


SUBJECT:  Research  of  A-E  Selection  Criteria 

(Ref.  my  30  Jan  87  letter) 

TO:  CES/CC 

1 .  This  letter  is  to  remind  you  of  my  request  in  the 
referenced  letter.  I  asked  that  you  forward  an  unused  copy 
of  the  evaluation  form,  or  scoresheet,  your  unit  uses  to 
select  A-E’s  for  the  design  of  construction  projects.  These 
forms  will  be  used  in  my  research  to  identify  trends  within 
and  across  major  commands  and  the  Air  Force.  1  ai so  none  to 
develop  a  consolidated  product  from  among  tne  inputs  of  a^x 
CONUS  bases. 

2.  Please  forward  your  unit's  contribution  to  my  research 
promptly  as  my  deadline  for  data  analysis  is  approaching.  I 
understand  your  own  press  with  deadlines,  but  I  would 
greatly  appreciate  your  participation  in  my  study. 

3.  If  you  have  already  sent  your  evaluation  form,  thank  you 
for  your  support.  If  not,  your  input  should  be  sent  to: 

1st  Lt  Mark  S.  Tissi 
AFIT/LSG 
WPAFB  OH  45433 

4.  Your  organization’s  input  will  be  kept  in  strictest 
confidence  throughout  the  research  effort.  Questions 
concerning  this  request  can  be  answered  by  calling  me  at 
AUTOVON  785-5435. 


MARK  S.  TISSI,  1st  Lt,  USAF 

Graduate  Student  in  Engineering  Management 

Air  Force  Institute  of  Technology 


Data  categorization  began  with  the  collection  of  the  62 


responses  to  the  letters  appearing  in  Appendix  B.  These 
responses  were  in  the  form  of  A-E  evaluation  scoresheets 
such  as  those  shown  in  Appendix  D.  Each  criterion,  and  a 
coded  symbol  of  the  base  which  used  it,  were  placed  in  one 
of  AFR  88-31 ’s  categories  of  criteria  (experience, 
capability,  workload,  location,  smali/minority  business, 
number  of  prior  DOD  awards,  and  dollar  value  of  prior 
awards),  or  they  were  placed  in  a  category  for  criteria  not 
fitting  any  of  the  seven  in  the  regulation. 

A  database  was  then  created  which  contained  each  base's 
criteria  and  the  maximum  possible  rating  for  each 
criterion.  Spreadsheets  were  then  created  from  the 
information  in  the  database  and  are  included  in  this 
appendix . 

Each  criterion’s  operational  definition  is  listed  below 
to  present  the  rationale  used  to  place  a  criterion  in  a 
particular  category.  The  column  labelled  "CODE"  contains 
the  symbol  for  the  base  whose  criteria  is  shown  in  the  row. 
The  remainder  of  the  columns  have  uppercase  alphabetic 
labels  which  are  keyed  to  the  operational  definitions  below. 

A.  General  Experience  -  the  listed  criterion  was 
no  more  specific  than  "Experience" 

B.  Specialized  Experience  -  experience  with  Air 
Force  projects  similar  to  that  to  be  awarded. 

C.  Past  Experience  -  quality  of  any  previous 
designs,  especially  those  done  for  the  Air 
Force . 
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Subtotal  of  Experience  Category  -  sum  of  the 
maximum  ratings  available  at  the  base. 

General  Capability  -  the  listed  criterion  was 
no  more  specific  than  "Capability" . 

Availability  of  Key  Personnel  -  the  employment 
of  a  complete  staff  of  architects  and 
engineers  of  each  required  discipline. 

Technical  Qualifications  -  professional 
registration  and  licensing  of  architects  and 
engineers . 

Subtotal  of  Capability  Category  -  sum  of  the 
maximum  ratings  available  at  the  base. 

General  Workload  -  the  listed  criterion  was  no 
more  specific  than  "Workload". 

Capacity  to  Do  Work  in  Required  Time  -  ability 
of  the  firm  to  meet  deadlines  with  available 
personnel . 

Subtotal  of  Workload  Category  -  sum  of  the 
maximum  ratings  available  at  the  base. 

Location  of  Firm  -  proximity  of  assigned 
designers  to  the  site  of  the  future 
construction . 

Small /Minority  Business  -  legal  qualification 
as  a  socially  and  economically  disadvantaged 
small  business  firm. 

Number  of  Prior  POD  Awards  -  DOD  projects 
previously  awarded  to  the  A-E  within  the  past 
three  fiscal  years.  Maximum  points  are 
assigned  to  firms  with  no  prior  awards. 

Dollar  Value  of  Prior  Awards  -  dollar  value  of 
awards  defined  in  N.  Maximum  points  are 
assigned  to  firms  with  no  prior  awards. 

Cost  Control  Effectiveness  -  ability  to 
deliver  required  work  within  the  budgeted 
amount . 

Innovative  Design  Capability  -  evidenced  by 
the  number  of  design  competitions  won  by  the 
firm’s  recent  work. 


R.  Construction  Inspection  Capability  -  the 
availability  of  qualified  inspectors  to 
oversee  construction  of  the  firm’s  design. 

S.  Organizational  Adequacy  -  general  assessment 
of  the  firm’s  organizational  structure. 

T .  Computer-aided  Design /Manufacturing  -  the 
access  of  the  firm  to  these  technological 
advances  in  facility  design. 

U.  Toll-free  Phone  -  the  firm  can  be  contacted  by 
phone  from  the  BCE  office  without  a  toll 
charge . 

Summary  statistics  computed  in  the  spreadsheets  induce 
TOTAL  POSSIBLE,  the  sum  of  the  maximum  possible  points 
available  from  all  criteria  used  by  the  base;  AVG,  the 
average  of  the  maximum  possible  points  among  bases  who  use 
the  criterion  represented  by  the  column;  and  PCT  USE,  the 
percentage  of  bases  using  the  criterion  represented  by  the 
column.  The  absence  of  a  percentage  indicates  a  100  percent 
usage  rate. 
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Appendix  D:  Representative  Responses  of  BCEs 

Sixty-two  Base  Civil  Engineers,  representing  77  percent 
of  those  solicited,  responded  to  the  letters  appearing  in 
Appendix  B.  All  of  the  responses  were  A-E  evaluation 
scoresheets  developed  by  the  base  for  preselection  and 
selection  boards.  Three  responses,  deliberately  chosen  to 
demonstrate  the  varying  degrees  of  detail  contained  in  all 
the  responses,  are  snowr.  in  cnis  appendix.  To  sat  e&a  a  r ::  *.  :.e 
anonymity  of  all  respondents,  any  references  to  the  source 
of  the  three  responses  shown  have  been  omitted. 


COO  to  5500,000  of  design  fees  foe  federal  projects 

5500,000  of  design  fees  far  federal  pro]®c:3  in  tne 

oriale:  Distribution  of  federal  design  contracts  among  many  A 

9ta!  Small  Disadvantaged _ 3u si  ness  Involvement  (form  254  ,  Item 


10  Points 


Ratine  Con s i de r a r i on 

a.  If  form  254  ,  Item  4  is  "yes'' 

b.  If  fora  254,  Item  4  is  "no"  or  n.mot  marked 

Rationale:  Oisadvantaged  involvement  must  count  a 

ave  9  rating  fa 

points.  0 i sadvan taged  firm  is  therefore  awarded  90/9  *  10  po; 
3.  Cost  Control  effectiveness-  10  Points 


Patino  Cons ider at  ion 

a.  Firm  has  demonstrated,  ability  to  design  within  the  ava 
funds  on  two  or  more  projects* within  past  12  months. 

b.  fits- has  demonstrated  ability  to  design  within  the  ava 
funds  on  one  project  within  past  12  months. 

c.  Ability  to  design  within  available  funds  has  not  been 
demonstrated. 

Pat  ionale :  decent  demonstrated  cost  control  effectiveness  on 
projects  is  required , 'to  insure  that  the  lirm  is  capaole  in  th 


on  this  oro-ec 


at  had 


:ot  ad 


Pat  ionale:  Bess  control  of  inspection  responsibilities  results 
tne  capaoility  is  in-house.  The  use  of  a  previously-hired 
consultant  for  this  purpose  is  considered  less  desiraole. 

1 0 .  Innovative  Design  Capability  -  10  Points 

Hating  Cons ider at  ion 

a.  firm  has  won  AF ,  TAC,  or  other  recognized  Professional 
Design  group/design  awards  within  last  two  years. 

b.  Firm  has  not  won  awards,  but  their  submittal  indicates 
successful  innovative  designs. 

c.  Neither  of  the  above. 

Rationale;  The  Air  Force  insists  on  design  excellence  -  awards 
generally  one  of  the  best  indicators  of  the  firm’s  innovative 
capability. 
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